Resource management for data streaming applications

This dissertation investigates novel middleware mechanisms for building streaming applications. Developing streaming applications is a challenging task because (i) they are continuous in nature; (ii) they require fusion of data coming from multiple sources to derive higher level information; (iii) they require efficient transport of data from/to distributed sources and sinks; (iv) they need access to heterogeneous resources spanning sensor networks and high performance computing; and (v) they are time critical in nature. My thesis is that an intuitive programming abstraction will make it easier to build dynamic, distributed, and ubiquitous data streaming applications. Moreover, such an abstraction will enable an efficient allocation of shared and heterogeneous computational resources thereby making it easier for domain experts to build these applications. In support of the thesis, I present a novel programming abstraction, called DFuse, that makes it easier to develop these applications. A domain expert only needs to specify the input and output connections to fusion channels, and the fusion functions. The subsystems developed in this dissertation take care of instantiating the application, allocating resources for the application (via the scheduling heuristic developed in this dissertation) and dynamically managing the resources (via the dynamic scheduling algorithm presented in this dissertation). Through extensive performance evaluation, I demonstrate that the resources are allocated efficiently to optimize the throughput and latency constraints of an application.Ph.D.Committee Chair: Ramachandran, Umakishore; Committee Member: Chervenak, Ann; Committee Member: Cooper, Brian; Committee Member: Liu, Ling; Committee Member: Schwan, Karste

Interactive Grid Architecture for Application Service Providers

This paper presents our Interactive Grid architecture for Application Service Providers (I-GASP). We envision IGASP as a solution for making computers available primarily for interactive use in a grid computing environment. A user might access such a computer for running diverse applications such as graphics rendering, scientific visualization or mechanical CAD. I-GASP consists of a grid middleware for provisioning these computers, remote display technology that goes across firewalls and several techniques for making the computers suitable for use in a grid environment, namely controlled shell and desktop, dynamic accounts, admission control, and monitoring and management agents. To minimize the amount of user data that needs to migrate to the assigned computer before an interactive session begins, we present our affinity scheduling algorithm that favors a computer where the user has previously had an interactive session

Automating Provisioning of Complete Software Stack in a Grid Environment

With scaling of data centers, clusters, and grids, it is going to be increasingly difficult to download, configure, install, update, and manage the software stack on the constituent nodes. Automation of this process is needed for the system to scale beyond a few hundred nodes. We present system architecture for a middleware service that performs this task in an automated manner. We propose that our architecture can be implemented as a grid service as part of the globus toolkit. Furthermore, we demonstrate the usefulness of the architecture through a prototype implementation which automates the whole process by integrating various existing solutions along with building intelligence into the middleware to handle such tasks

Resource Allocation for Remote Desktop Sessions in Utility Grids

Emerging large scale utility computing systems such as Grids promise computing and storage to be provided to end users as a utility. System management services deployed in the middleware are a key to enabling this vision. Utility Grids provide a challenge in terms of scale, dynamism, and heterogeneity of resources and workloads. In this paper, we present a model based architecture for resource allocation services for Utility Grids. The proposed service is built in the context of interactive remote desktop session workloads and takes application performance QoS models into consideration. The key design guidelines are hierarchical request structure, application performance models, remote desktop session performance models, site admission control, multi-variable resource assignment system, and runtime session admission control. We have also built a simulation framework that can handle mixed batch and remote desktop session requests, and have implemented our proposed resource allocation service into the framework. We present some results from experiments done using the framework. Our proposed architecture for resource allocation services addresses the needs of emerging utility computing systems and captures the key concepts and guidelines for building such services in these environments

Resource Allocation for Remote Desktop Sessions in Utility Grids

Emerging large scale utility computing systems such as Grids promise computing and storage to be provided to end users as a utility. System management services deployed in the middleware are a key to enabling this vision. Utility Grids provide a challenge in terms of scale, dynamism, and heterogeneity of resources and workloads. In this paper, we present a model based architecture for resource allocation services for Utility Grids. The proposed service is built in the context of interactive remote desktop session workloads and takes application performance QoS models into consideration. The key design guidelines are hierarchical request structure, application performance models, remote desktop session performance models, site admission control, multi-variable resource assignment system, and runtime session admission control. We have also built a simulation framework that can handle mixed batch and remote desktop session requests, and have implemented our proposed resource allocation service into the framework. We present some results from experiments done using the framework. Our proposed architecture for resource allocation services addresses the needs of emerging utility computing systems and captures the key concepts and guidelines for building such services in these environments

Computing, remote

desktop sessions

DFuse: A Framework for Distributed Data Fusion

Simple in-network data aggregation (or fusion) techniques for sensor networks have been the focus of several recent research efforts, but they are insufficient to support advanced fusion applications. We extend these techniques to future sensor networks and ask two related questions: (a) what is the appropriate set of data fusion techniques, and (b) how do we dynamically assign aggregation roles to the nodes of a sensor network ? We have developed an architectural framework, DFuse, for answering these two questions. It consists of a data fusion API and a distributed algorithm for energy-aware role assignment. The fusion API enables an application to be specified as a coarsegrained dataflow graph, and eases application development and deployment. The role assignment algorithm maps the graph onto the network, and optimally adapts the mapping at run-time using role migration. Experiments on an iPAQ farm show that the fusion API has low-overhead, and the role assignment algorithm with role migration significantly increases the network lifetime compared to any static assignment